1. Learning Objectives

• Differentiate between qualitative and quantitative data within a research framework.
• Identify and apply appropriate primary and secondary data collection techniques.
• Evaluate sampling strategies and justify their suitability for specific problems.
• Describe the instruments used for data collection and their reliability and validity considerations.
• Apply descriptive and inferential statistical techniques.
• Interpret analytical results in the context of research objectives and hypotheses.
• Recognise the role of instrumentation and modern computational tools in the research workflow.

2. The Nature and Classification of Research Data

2.1 Quantitative Data and Measurement Scales

Quantitative data are expressed numerically and can be subjected to arithmetic operations and statistical analysis. In computing, examples include algorithm response times in milliseconds, memory consumption in megabytes, protocol throughput in packets per second, and classifier accuracy as a percentage. They are further classified as discrete (countable, often integers — the number of concurrent users on a web server) or continuous (any value within a range — execution time, signal strength).

2.2 Qualitative and Mixed Data

Qualitative data describe characteristics, perceptions or experiences in non-numeric form: interview transcripts, open-ended survey responses, observational notes, case study narratives. They are indispensable in human–computer interaction, software engineering practice research and the study of ethical implications of AI. Qualitative data may be textual, visual or auditory, and typically require coding and thematic interpretation rather than statistical summarisation.

Mixed data integrate numerical measurement with contextual qualitative insight to provide a richer understanding. This methodological pluralism is increasingly common in design science research and user studies.

3. Sources and Methods of Data Collection

3.1 Primary Data Collection

Primary data are collected directly by the researcher for the problem at hand. They offer high relevance but require greater investment in time, cost and effort.

• Surveys and questionnaires. A structured or semi-structured set of questions administered to a sample, distributed physically, telephonically or online. Best practice: design clear, unambiguous, bias-free items; combine closed- and open-ended questions; pilot test to refine the instrument; use digital platforms such as Google Forms, Qualtrics or SurveyMonkey for efficient capture.
• Interviews. Structured (predetermined questions in fixed sequence), semi-structured (core questions with freedom to probe) or unstructured (open conversation). Particularly valuable for understanding developer experiences or end-user perceptions of security mechanisms.
• Observation. Systematic recording of behaviour as it occurs, either participant (the researcher is part of the group) or non-participant (the researcher remains external). Applied to user interactions with software, network traffic patterns, or laboratory protocol execution.
• Experiments. Manipulation of independent variables to observe effects on dependent variables under controlled conditions — for example evaluating a sorting algorithm across varying input sizes and distributions. The design specifies the independent variable (algorithm type), the dependent variable (execution time) and the control variables held constant.
• Case studies. An intensive, holistic description and analysis of a single instance or bounded system, such as the adoption of a software framework within an organisation. May be exploratory, descriptive or explanatory.
• Focus groups. A small selected group discussing a topic under a moderator. The interactive dialogue reveals collective perspectives and social dynamics not accessible through individual interviews.

3.2 Secondary Data Collection

Secondary data consist of information already gathered and recorded by others:

• published academic literature — journals, conference proceedings, theses;
• government and industry reports;
• public datasets such as the UCI Machine Learning Repository, Kaggle, and governmental open-data portals;
• internal organisational records — logs, performance metrics, audit trails;
• digital archives, online encyclopedias and patent databases.

Secondary data are cost-effective and time-efficient, but the researcher must evaluate their relevance, accuracy, timeliness and methodological soundness before use.

4. Sampling Techniques

Because it is rarely feasible to examine every element of a population, researchers rely on samples. The validity of findings depends critically on the sampling strategy.

4.1 Probability Sampling

Each member of the population has a known, non-zero chance of selection.

• Simple random sampling — every individual chosen purely by chance, by lottery method or random number generator.
• Systematic sampling — every kth element selected from an ordered list.
• Stratified sampling — the population is divided into homogeneous subgroups (strata) and random samples drawn from each in proportion to its size.
• Cluster sampling — the population is divided into clusters, often geographic, and entire clusters are randomly selected.

4.2 Non-Probability Sampling

Selection rests on the researcher's judgement or convenience rather than random assignment.

• Convenience sampling — respondents selected because they are readily accessible.
• Purposive (judgmental) sampling — participants chosen for specific characteristics relevant to the study.
• Snowball sampling — existing participants refer additional subjects; useful for hidden or specialised populations.
• Quota sampling — categories are pre-defined and participants selected non-randomly to fill them.

4.3 Determining Sample Size

Sample-size calculations balance statistical power, confidence level, acceptable margin of error and population variability. A frequently used formula for estimating a proportion is:

where Z is the standard normal value for the desired confidence level (1.96 for 95% confidence), p is the estimated proportion (commonly 0.5 for maximum variability), and e is the acceptable margin of error.

5. Instrumentation, Reliability and Validity

5.1 Measurement Instruments

A research instrument is the tool used to capture observations. The principal instruments in computing research are:

• Questionnaires and interview protocols — standardised forms for eliciting self-reported data.
• Sensors and loggers — hardware or software devices that automatically record events: application performance monitors, network analysers such as Wireshark.
• Profiling and benchmarking tools — perf, gprof, JMH, recording execution times, memory usage and resource consumption.
• Eye-trackers and biosensors — used in HCI research to capture gaze, heart rate or galvanic skin response.
• Source code analysers — SonarQube, PMD, measuring cyclomatic complexity, lines of code and defect density.

5.2 Reliability and Validity

The credibility of collected data rests on two psychometric properties.

Reliability is the consistency of measurements across time, instruments or raters. The most common coefficient is Cronbach's alpha:

where k is the number of items, σ²_Yi the variance of item i, and σ²_X the total variance. Values of α ≥ 0.7 are generally considered acceptable.

Validity is the degree to which an instrument measures what it purports to measure:

• Content validity — coverage of the full domain by the instrument.
• Construct validity — alignment with theoretical expectations.
• Criterion-related validity — correspondence with an external benchmark.

Pre-testing. Before deployment, instruments should be piloted on a small sample to identify ambiguities, technical errors and timing issues. Feedback from the pilot refines the instrument for the main study.

6. Data Preparation

Raw data are rarely suitable for immediate analysis. Preparation involves six steps:

1. Editing — detecting and correcting errors, omissions and inconsistencies.
2. Coding — assigning numerical or symbolic labels to responses, especially for qualitative and categorical data.
3. Data entry — transferring data into digital formats, often spreadsheets or statistical software.
4. Transcription — converting audio recordings of interviews into text.
5. Cleaning — handling missing values through imputation or exclusion, detecting outliers, standardising formats.
6. Tabulation — organising data into structured tables for inspection.

7. Data Analysis

7.1 Quantitative Analysis

Descriptive statistics summarise the salient features of a dataset. Measures of central tendency are the mean, median and mode:

Measures of dispersion are the range, variance and standard deviation:

Measures of shape — skewness and kurtosis — describe asymmetry and peakedness, and are complemented by frequency distributions and visual representations: histograms, box plots, scatter plots and bar charts.

Inferential statistics allow generalisation from a sample to a broader population. Hypothesis testing formulates a null hypothesis H₀ and an alternative H₁, then computes a test statistic and a corresponding p-value.

• t-tests compare two group means.
• Analysis of variance (ANOVA) compares three or more group means.
• Chi-square examines associations between categorical variables.
• Correlation and regression quantify relationships among variables.
• Non-parametric tests — Mann–Whitney U, Wilcoxon signed-rank, Kruskal–Wallis — are used where data violate normality assumptions.

7.2 Qualitative Analysis

• Thematic analysis — identifying, analysing and reporting patterns (themes) within data.
• Content analysis — coding textual material into categories and quantifying occurrences.
• Discourse analysis — examining language use in social context.
• Grounded theory — building theory inductively from data through open, axial and selective coding.
• Narrative analysis — focusing on the stories people tell and their structural and linguistic features.

Software tools that assist qualitative analysis include NVivo, ATLAS.ti and MAXQDA.

7.3 Mixed-Methods Analysis

Mixed-methods studies integrate quantitative and qualitative strands. Analysis proceeds in parallel, with results triangulated to enrich interpretation. Common designs are concurrent triangulation, sequential explanatory and sequential exploratory.

8. Interpretation of Data

Interpretation is the intellectual bridge between analysis and conclusion. It involves relating findings to the hypotheses, questions or objectives; comparing results with prior studies and theoretical expectations; identifying patterns, anomalies and unexpected observations; acknowledging limitations in data, methods and generalisability; and offering explanations and implications.

9. Worked Illustration: Sorting Algorithm Performance

10. Common Errors in Data Collection and Analysis

• Selection bias — systematic differences between sample and population.
• Measurement bias — systematic distortion in instruments or procedures.
• Non-response bias — failure to capture data from significant segments of the sample.
• Confirmation bias — interpreting ambiguous evidence in favour of one's own hypothesis.
• p-hacking — performing many tests and selectively reporting only those that reach significance.
• Violation of assumptions — applying parametric tests to non-normal data without transformation or a non-parametric alternative.

11. Summary

This lecture surveyed the principal elements of data collection and analysis: the distinction between qualitative and quantitative data, primary and secondary sources, sampling strategies, measurement instruments, and the psychometric criteria of validity and reliability. It then addressed data preparation, descriptive and inferential statistical methods, qualitative analytical techniques, and the interpretive practices that culminate in research conclusions. Mastery of these methods equips the computer science researcher to generate evidence that is both credible and meaningful.

12. Practice and Review Questions

1. Differentiate between primary and secondary data, providing two computing examples of each. 4 marks
2. A researcher wishes to estimate the average time students spend daily on programming practice, with a margin of error of ±10 minutes at 95% confidence. Assuming a population standard deviation of 30 minutes, compute the required sample size using n = ( Z · σ / e )². 5 marks
3. Explain the difference between reliability and validity. Why is a measure that is reliable not necessarily valid? 5 marks
4. Describe three situations in which qualitative methods are preferable to quantitative methods in computing research. 6 marks
5. A study compares the execution times of three sorting algorithms across 50 test cases each. Which statistical test is most appropriate, and why? State the null and alternative hypotheses. 6 marks
6. What is stratified sampling? Under what circumstances is it more appropriate than simple random sampling? 5 marks
7. Discuss three common errors in data analysis and how they may be mitigated. 6 marks
8. Describe the role of pilot testing in questionnaire design. 4 marks

13. Suggested Readings

• Creswell, J. W., & Creswell, J. D. (2018). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (5th ed.). SAGE.
• Kothari, C. R. (2004). Research Methodology: Methods and Techniques (2nd ed.). New Age International.
• Field, A. (2018). Discovering Statistics Using IBM SPSS Statistics (5th ed.). SAGE.
• Bryman, A. (2016). Social Research Methods (5th ed.). Oxford University Press.
• Wohlin, C., et al. (2012). Experimentation in Software Engineering. Springer.